Rolling Gesture Detection Using an Electronic Device

ABSTRACT

An electronic device with one or more processors and memory detects a button press of a respective button of a plurality of buttons that include a first button that corresponds to a first type of operation and a second button that corresponds to a second type of operation. The device determines, in conjunction with detecting the button press, a rolling gesture metric corresponding to performance of a rolling gesture comprising rotation about a longitudinal axis of the electronic device. After determining the rolling gesture metric, when the respective button is the first button, the device initiates performance, in a respective user interface, of an operation of the first type in accordance with the rolling gesture metric and when the respective button is the second button, the device initiates performance, in the respective user interface, of an operation of the second type in accordance with the rolling gesture metric.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.12/984,547 filed Jan. 4, 2011, entitled “Rolling Gesture Detection Usinga Multi-Dimensional Pointing Device,” which claims priority under 35U.S.C. §119 to U.S. Provisional Patent Application No. 61/292,797 filedJan. 6, 2010, entitled “Rolling Gesture Detection Using aMulti-Dimensional Pointing Device,” all of which are incorporated byreference herein in their entireties.

This application is related to pending U.S. patent application Ser. No.12/436,717 filed on May 6, 2009, entitled “System and Method forDetermining an Attitude of a Device Undergoing Dynamic AccelerationUsing a Kalman Filter,” which claims priority under 35 U.S.C. §119 toU.S. Provisional Patent Application No. 61/143,133 filed Jan. 7, 2009,entitled “System and Method for Determining an Attitude of a DeviceUndergoing Dynamic Acceleration Using Kalman Filter,” both of which areincorporated by reference herein in their entireties.

This application is also related to U.S. patent application Ser. No.12/338,991 filed on Dec. 18, 2008, entitled “System and Method forDetermining an Attitude of a Device Undergoing Dynamic Acceleration,”which is incorporated by reference herein in its entirety.

This application is also related to U.S. patent application Ser. No.12/338,996 filed on Dec. 18, 2008, entitled “Host System and Method forDetermining an Attitude of a Device Undergoing Dynamic Acceleration,”which is incorporated by reference herein in its entirety.

TECHNICAL FIELD

The disclosed embodiments relate generally to detecting performance of arolling gesture using a multi-dimensional pointing device and conveyinga corresponding rolling gesture metric to a host system.

BACKGROUND

A pointing device (e.g., a mouse, a trackball, etc.) may be used tointeract with objects within a user interface of a computer system orother electronic devices (e.g., a set top box, etc.). Existing pointingdevices offer a limited set of user interface operations. For example, amouse is typically moved across a flat surface to produce translationalmovement (e.g., in the x and y directions) of objects in the userinterface of a computer system. Another type of pointing device is afree space pointer. The free space pointer is typically moved in threedimensions. However, like the mouse, the free space pointer is limitedto producing translational movement of objects in the user interface ofthe computer system, for example by pointing to an object and thenmoving the pointer to indicate a new position to which the object is tobe moved.

SUMMARY

Some embodiments provide a system, a computer readable storage mediumincluding instructions, and a computer-implemented method for detectingperformance of a rolling gesture using a multi-dimensional pointingdevice. An initiation of a gesture by a user of the multi-dimensionalpointing device is detected. A rolling gesture metric corresponding toperformance of a rolling gesture comprising rotation of themulti-dimensional pointing device about a longitudinal axis of themulti-dimensional pointing device is determined. Informationcorresponding the rolling gesture metric is conveyed to a clientcomputer system, wherein the client computer system is configured tomanipulate an object in a user interface of the client computer systemin accordance with the rolling gesture metric.

In some embodiments, the initiation of the gesture by the user of themulti-dimensional pointing device is detected by detecting the pressingof a button on the multi-dimensional pointing device.

In some embodiments, the button is selected from the group consisting ofa volume button, a channel button, a video input button, an audio inputbutton, and a gesture button.

In some embodiments, the rolling gesture metric corresponds to a changein attitude of the pointing device upon initiation of the rollinggesture.

In some embodiments, the rolling gesture metric is selected from thegroup consisting of a roll angle, a roll rate, a roll acceleration(i.e., a time derivative of the roll rate), and a predefined combinationof two or more of the roll angle, roll rate and roll acceleration.

In some embodiments, the corresponding rolling gesture metric isdetermined as follows. A change in attitude of the multi-dimensionalpointing device, corresponding to rotation about a longitudinal axis ofthe multi-dimensional pointing device, is calculated based on one ormore accelerometer measurements from one or more multi-dimensionalaccelerometers of the multi-dimensional pointing device and one or moremagnetic field measurements from one or more multi-dimensionalmagnetometers of the multi-dimensional pointing device. The rollinggesture metric is then calculated based on the change in attitude of themulti-dimensional pointing device.

In some embodiments, the corresponding rolling gesture metric isdetermined as follows. A change in attitude of the multi-dimensionalpointing device is calculated based on one or more accelerometermeasurements from one or more multi-dimensional accelerometers of themulti-dimensional pointing device and one or more magnetic fieldmeasurements from one or more multi-dimensional magnetometers of themulti-dimensional pointing device. It is then determined that themulti-dimensional pointing device is undergoing a rotation about alongitudinal axis of the multi-dimensional pointing device based on thechange in attitude of the multi-dimensional pointing device. The rollinggesture metric is then calculated based on the change in attitude of themulti-dimensional pointing device.

In some embodiments, the rolling gesture is mapped to a scrollingoperating that is performed on the object in the user interface of theclient computer system. For example, the object on which the scrollingoperation is performed may be selected from the group consisting of aweb page, a document, and a list.

In some embodiments, the rolling gesture is mapped to a rotationoperation that is performed on the object in the user interface of theclient computer system. For example, the object on which the rotationoperation is performed may be selected from the group consisting of adial, a photograph, and a page of a document.

In some embodiments, the rolling gesture metric is mapped to a number ofclicks of a mouse wheel over a time interval.

In some embodiments, detecting initiation of the gesture includesreceiving a message from the client computer system that indicates thatthe user of the multi-dimensional pointing device selected a userinterface element (e.g., a menu item, an icon, etc.) in the userinterface of the client computer system that initiates the detection ofgestures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary multi-dimensional pointing devicecoupled to an exemplary host system through a wireless interface,according to some embodiments.

FIG. 2 is a block diagram illustrating an exemplary multi-dimensionalpointing device, according to some embodiments.

FIG. 3 is a block diagram illustrating inputs, outputs, and operationsof an exemplary software architecture for a host system, according tosome embodiments.

FIG. 4 is a block diagram illustrating an exemplary device-side firmwarefor a multi-dimensional pointing device, according to some embodiments.

FIG. 5 is a diagram illustrating exemplary gravity and magnetic fieldvectors that can be used to determine attitude, according to someembodiments.

FIG. 6 is a diagram illustrating an attitude determination error causedat least in part by dynamic acceleration, according to some embodiments.

FIG. 7 is a diagram illustrating an exemplary technique for compensatingfor dynamic acceleration in attitude calculations, according to someembodiments.

FIG. 8 is a block diagram illustrating an exemplary method fordetermining an attitude of a device undergoing dynamic acceleration,according to some embodiments.

FIG. 9 is a graph illustrating an exemplary quaternion, according tosome embodiments.

FIG. 10 is a flow diagram of a method for determining an attitude of adevice undergoing dynamic acceleration, according to some embodiments.

FIG. 11 presents a block diagram of an exemplary multi-dimensionalpointing device, according to some embodiments.

FIG. 12 presents a block diagram of an exemplary host system, accordingto some embodiments.

FIG. 13 presents a block diagram illustrating an exemplarymulti-dimensional pointing device performing a rolling gesture in anexemplary user interface of an exemplary host system, according to someembodiments.

FIG. 14 presents a block diagram illustrating parameters that aretransmitted between a multi-dimensional pointing device and a hostsystem, according to some embodiments.

FIG. 15 is a flow diagram of a method for detecting performance of arolling gesture using a multi-dimensional pointing device, according tosome embodiments.

FIG. 16 is a flow diagram of a method for calculating a rolling gesturemetric, according to some embodiments.

FIG. 17 is a block diagram illustrating axes of rotation for amulti-dimensional pointing device, according to some embodiments.

Like reference numerals refer to corresponding parts throughout thedrawings.

DESCRIPTION OF EMBODIMENTS Digital Convergence

Before discussing embodiments that can be used to solve theaforementioned problems, it is instructive to discuss the possible usesof the embodiments described herein. The idea of “digital convergence”has been a prevalent pursuit for many years. One aspect of “digitalconvergence” is making content (e.g., digital content) available to auser on any type of display device. The struggle towards digitalconvergence is particularly acute among personal computer (PC)manufacturers, broadband media providers, and consumer electronics (CE)manufacturers.

CE manufacturers and broadband media providers have experienced theeffects of the rise of Internet-distributed content (e.g., digital movieand music downloads, etc.), which have diverted consumers from theirproducts and services. Accordingly, consumers spend more time in frontof their personal computers (PCs). Digital convergence may allow CEmanufacturers and broadband media providers to recapture consumerattention by routing content consumption through their domains (e.g.,cable and satellite transmissions to a television set).

Unfortunately, one substantial hurdle to digital convergence is the lackof an advanced user interface for the television set (or other CEdevices). Although high-definition television (HDTV) has increased theresolution of the television programs displayed, the remote control of atelevision set or a cable/satellite set-top box (STB) remains archaic:including a numeric keypad, up/down/left/right arrows, and a largenumber of predefined function keys. This lack of an advanced userinterface makes the PC a logical venue for interactive content.

Digital convergence may redefine the role of the television set. Insteadof just providing multimedia content for passive consumption, thetelevision set may be a center of interactivity, providing access tophotos, movies, music, games, phone calls, video conferences, etc.However, to facilitate the goal of digital convergence, an advanced userinterface must be provided for the television set. Accordingly, thesimple remote controller for existing television sets must be replacedwith a device that can interact with the advanced user interface.Furthermore, the remote controller must remain cost-effective (e.g.,less than $10), must have long battery life, and must be responsive touser input.

Multi-Dimensional Pointing Device

A multi-dimensional pointing device may be used to interact withadvanced user interfaces that are needed to achieve digital convergence.FIG. 1 illustrates an exemplary multi-dimensional pointing (MDP) device102 coupled to an exemplary host system 101 through a wirelessinterface, according to some embodiments. In these embodiments, a user103 can use the multi-dimensional pointing device 102 to issue commandsto the host system 101, control objects in the user interface of thehost system 101, and/or position objects in the user interface of thehost system 101. In some embodiments, the multi-dimensional pointingdevice 102 is sensitive to six degrees of freedom: x, y, z, yaw, pitch,and roll.

In some embodiments, the wireless interface is selected from the groupconsisting of: a Wi-Fi interface, a Bluetooth interface, an infraredinterface, an audio interface, a visible light interface, a radiofrequency (RF) interface, and any combination of the aforementionedwireless interfaces.

In some embodiments, data (e.g., raw measurements, calculated attitude,correction factors, position information, etc.) from themulti-dimensional pointing device 102 is received and processed by ahost side device driver on the host system 101. The host system 101 canthen use this data to position cursors, objects, etc., in the userinterface of the host system 101.

In some embodiments, the wireless interface is a unidirectional wirelessinterface from the multi-dimensional pointing device to the host system101. In some embodiments, the wireless interface is a bidirectionalwireless interface. In some embodiments, bidirectional communication isused to perform handshaking and pairing operations.

In some embodiments, a wired interface can be used instead of a wirelessinterface. As with the wireless interface, the wired interface may be aunidirectional or bidirectional wired interface.

As mentioned above, the act of moving a multi-dimensional pointingdevice around creates accelerations and decelerations that may causeconventional attitude-determination techniques to fail. Specifically,consider a device that includes a single multi-dimensional magnetometer(e.g., a tri-axial magnetometer) and a single multi-dimensionalaccelerometer (e.g., a tri-axial accelerometer), which is subject todynamic acceleration. Note that the term “dynamic acceleration” refersto acceleration and/or deceleration (e.g., accelerations/decelerationsduring movement of the device). Applying the TRIAD technique to magneticfield measurements from a single multi-dimensional magnetometer andacceleration measurements from a single multi-dimensional accelerometerresults in attitude measurements that include errors. The errors arisebecause the TRIAD technique depends on a constant relationship betweenthe Earth's magnetic field and gravity. Consequently, the TRIADtechnique only produces correct attitude measurements when the device isnot undergoing dynamic acceleration (e.g., at rest or at constantvelocity). If the device is being accelerated, the accelerationmeasurement includes a combination of gravity and the accelerationimparted by movements of the device. Using this acceleration measurementto represent the Earth's gravity produces substantial errors in thecomputed attitude. These problems are described in more detail withrespect to FIGS. 5-7 below.

One solution is to use a multi-dimensional pointing device that includesa gyroscope (e.g., a MEMS gyroscope). However, the physics of thegyroscopes can cause artifacts. For example, these types ofmulti-dimensional pointing devices can drift when the device is held ina stationary position. Furthermore, these multi-dimensional pointingdevices can require substantial force before the device produces areaction in the user interface.

Thus, to solve the aforementioned problems, some embodiments usemagnetic field measurements from one or more multi-dimensionalmagnetometers and acceleration measurements from two or moremulti-dimensional accelerometers that are included in amulti-dimensional pointing device to calculate the attitude of thedevice. In these embodiments, the calculated attitude of themulti-dimensional pointing device is compensated for errors that wouldotherwise be caused by dynamic acceleration. In some embodiments, themulti-dimensional accelerometers are placed a specified distance apartin a rigid frame (e.g., a printed circuit board on the device). When themulti-dimensional pointing is rotated, the multi-dimensionalaccelerometers experience different accelerations due to their differentradiuses of rotation. Note that when the frame is moved in translation(e.g., without rotation), all the accelerometers experience the sameacceleration. It is then possible to use the differences in theaccelerometer readings to distinguish between user movement (e.g.,dynamic acceleration) and the acceleration caused by Earth's gravity tocorrectly estimate the attitude of the device.

FIG. 2 is a block diagram illustrating an exemplary multi-dimensionalpointing device 200, according to some embodiments. Themulti-dimensional pointing (MDP) device 200 may be the multi-dimensionalpointing device 102 in FIG. 1. The multi-dimensional pointing device 200includes two or more multi-dimensional accelerometers 201-202 thatproduce composite acceleration measurements 204-205 (e.g., acomposite/vector sum of translational acceleration vectors 210,rotational acceleration vectors 211-212, and acceleration due to Earth'sgravity), one or more multi-dimensional magnetometers 203 that producemagnetic field measurements 206 (e.g., the Earth's magnetic field),buttons 207, and a power supply and/or battery 208. In some embodiments,the two or more multi-dimensional accelerometers 201-202 that produceacceleration measurements 204-205, one or more multi-dimensionalmagnetometers 203 that produce the magnetic field measurements 206,buttons 207, and the power supply or battery 208 are all enclosed in ahousing 209 of the multi-dimensional pointing device 200.

In some embodiments, the two or more multi-dimensional accelerometers201-202 are selected from the group consisting of: a 2-axisaccelerometer that measures a magnitude and a direction of anacceleration force in two dimensions and a 3-axis accelerometer thatmeasures a magnitude and a direction of an acceleration force in threedimensions.

In some embodiments, the one or more multi-dimensional magnetometers 203are selected from the group consisting of: a 2-axis magnetometer thatmeasures a magnitude and a direction of a magnetic field in twodimensions and a 3-axis magnetometer that measures a magnitude and adirection of a magnetic field in three dimensions.

In some embodiments, the multi-dimensional pointing device 200 alsoincludes one or more of the following additional user interfacecomponents: a keypad, one or more thumb wheels, one or morelight-emitting diodes (LEDs), a audio speaker, an audio microphone, aliquid crystal display (LCD), etc.

In some embodiments, the multi-dimensional pointing device 200 includesone or more processors (e.g., 1102, FIG. 11). In these embodiments, theone or more processors process the acceleration measurements receivedfrom the multi-dimensional accelerometers 201-202 and/or magnetic fieldmeasurements received from the multi-dimensional magnetometer 203 todetermine displacements (e.g., lateral displacements and/or attitudechanges) of the multi-dimensional pointing device 200. Thesecalculations are described in more detail with respect to FIGS. 8-12below.

In some embodiments, the one or more processors of the multi-dimensionalpointing device 200 perform one or more of the following operations:sampling measurement values, at a respective sampling rate, produced byeach of the multi-dimensional accelerometers 201-202 and themulti-dimensional magnetometers 203; processing sampled data todetermine displacement; transmitting displacement information to thehost system 101; monitoring the battery voltage and alerting the hostsystem 101 when the charge of the battery is low; monitoring other userinput devices (e.g., keypads, buttons, etc.), if any, on themulti-dimensional pointing device 200; continuously or periodically runbackground processes to maintain or update calibration of themulti-dimensional accelerometers 201-202 and the multi-dimensionalmagnetometers 203; provide feedback to the user as needed on the remote(e.g., via LEDs, etc.); and recognizing gestures performed by usermovement of the multi-dimensional pointing device 200.

Software Architecture

FIG. 3 is a block diagram illustrating an exemplary softwarearchitecture 300 for the host system 101. The software architecture 300includes a monitor application 301 to receive either accelerometer andmagnetometer measurements or attitude measurements from themulti-dimensional pointing device 200, depending on whether themulti-dimensional pointing device 200 or the host system processes themeasurements so as to produce attitude measurements. The softwarearchitecture also includes a program/file directory 302 (e.g., anelectronic program guide, etc.) that includes information about programsand/or media files (e.g., titles, times, channels, etc.), avideo-on-demand application 303 that provides access to one or morevideo-on-demand services, online applications 304 that provide access toapplications provided by a service provider (e.g., cable/satellitetelevision providers, Internet service providers, Internet websites,game providers, online multimedia providers, etc.), and terminal basedapplications 305 that are (or that provide access to) applications thatare resident on the host system 101 (e.g., games that are played on thehost system, Internet browsing applications, multimedia viewing and/orsharing applications, email applications, etc.). In some embodiments,the multi-dimensional pointing device 200 includes a subset of theseapplications. Furthermore, the multi-dimensional pointing device 200 mayinclude additional applications, modules and data structures notdescribed above.

The software architecture 300 also includes an operating system (e.g.,OpenCable Application Platform (OCAP), Windows, Linux, etc.) 310, whichincludes an execution engine (or virtual machine) 311 that executesapplications, an optional API 312 for communicating with amulti-dimensional pointing device that does not conform to a humaninterface standard implemented in the operating system 310, middleware313 that provides management of the resources of the host system 101(e.g., allocation of memory, access to access hardware, etc.) andservices that connect software components and/or applications,respectively, and host device drivers 314. In some embodiments, the hostdevice drivers 314 adjust the gain of the multi-dimensional pointingdevice 102 based on the resolution and/or aspect ratio of the display ofthe host system 101, translates physical movement of themulti-dimensional pointing device 102 to movement of a cursor (or anobject) within the user interface of the host system 101, allows hostapplications to adjust cursor movement sensitivity, and/or reportshardware errors (e.g., a battery low condition, etc.) to the middleware313.

In some embodiments, the multi-dimensional pointing device 102periodically samples its sensors. The multi-dimensional pointing device102 may also periodically provide the sampled sensor data to the hostsystem 101 at a respective update rate. To reduce power consumptioncaused by transmitting data to the host system 101, the update rate maybe set at a substantially smaller rate than the sampling rate. Note thatthe minimum update rate may be governed by the frame rate of the displayof the host system (e.g., 25 Hz in Europe and 30 Hz in the United Statesand Asia). Note that there may be no perceivable advantage in providingfaster updates than the frame rate except when the transmission media islossy.

In some embodiments, the multi-dimensional pointing device 102 usesdigital signal processing techniques. Thus, the sampling rate must beset high enough to avoid aliasing errors. Movements typically occur ator below 10 Hz, but AC power can create ambient magnetic fieldfluctuations at 50-60 Hz that can be picked up by a magnetometer. Forexample, to make sure there is sufficient attenuation above 10 Hz, themulti-dimensional pointing device 102 may use a 100 Hz sampling rate anda 50 Hz update rate.

In some embodiments, the multi-dimensional pointing device 102 reportsraw acceleration and magnetic field measurements to the host system 101.In these embodiments, the host device drivers 314 calculate lateraland/or angular displacements based on the measurements. The lateraland/or angular displacements are then translated to cursor movementsbased on the size and/or the resolution of the display of the hostsystem 101. In some embodiments, the host device drivers 314 use adiscrete representation of angular displacement to perform sampling rateconversion to smoothly convert from the physical resolution of themulti-dimensional pointing device 102 (e.g., the resolution of theaccelerometers and/or the magnetometers) to the resolution of thedisplay.

In some embodiments, the host device drivers 314 interpret a sequence ofmovements (e.g., changes in attitude, displacements, etc.) as a gesture.For example, the user 103 may use the multi-dimensional pointing device102 to move a cursor in a user interface of the host system 101 so thatthe cursor points to a dial on the display of the host system 101. Theuser 103 can then select the dial (e.g., by pressing a button on themulti-dimensional pointing device 102) and turn the multi-dimensionalpointing device 102 clockwise or counter-clockwise (e.g., roll) toactivate a virtual knob that changes the brightness, contrast, volume,etc., of a television set. Thus, the user 103 may use a combination orsequence of keypad presses and pointing device movements to conveycommands to the host system. Similarly, the user 103 may use a twist ofa wrist to select the corner of a selected image (or video) for sizingpurposes. Note that the corner of an image may be close to anotheractive object. Thus, selecting the image may require carefulmanipulation of the multi-dimensional pointing device 102 and could be atiresome exercise. In these cases, using a roll movement as a contextsensitive select button may reduce the accuracy users need to maintainwith the movement of the multi-dimensional pointing device 102.

In some embodiments, the multi-dimensional pointing device 102 computesthe physical displacement of the device and transmits the physicaldisplacement of the device to the host system 101. The host devicedrivers 314 interpret the displacement as cursor movements and/orgestures. Thus, the host device drivers 314 can be periodically updatedwith new gestures and/or commands to improve user experience withouthaving to update the firmware in the multi-dimensional pointing device102.

In some other embodiments, the multi-dimensional pointing device 102computes the physical displacement of the device and interprets thedisplacements as cursor movements and/or gestures. The determined cursormovements and/or gestures are then transmitted to the host system 101.

In some embodiments, the multi-dimensional pointing device 102 reportsits physical spatial (e.g., lateral and/or angular) displacements basedon a fixed spatial resolution to the host system 101. The host devicedrivers 314 interpret the distance and/or angle traversed intoappropriate cursor movements based on the size of the display and/or theresolution of the display. These calculated displacements are thentranslated into cursor movements in the user interface of the hostsystem 101.

Although the multi-dimensional pointing device 102 may provide data(e.g., position/displacement information, raw measurements, etc.) to thehost system 101 at a rate greater than the frame rate of a display ofthe host system 101, the host device drivers 314 needs to be robustenough to accommodate situations where packet transmission fails. Insome embodiments, each packet received from the multi-dimensionalpointing device 102 is time stamped so that the host device drivers 314can extrapolate or interpolate missing data. This time stamp informationmay also be used for gesture recognition to compensate for a lossytransmission media.

In some embodiments, the multi-dimensional pointing device 102 omitspackets to conserve power and/or bandwidth. In some embodiments, themulti-dimensional pointing device 102 omits packets to conserve powerand/or bandwidth only if it is determined that the host device drivers314 can recreate the lost packets with minimal error. For example, themulti-dimensional pointing device 102 may determine that packets may beomitted if the same extrapolation algorithm is running on the hostsystem 101 and on the multi-dimensional pointing device 102. In thesecases, the multi-dimensional pointing device 102 may compare the realcoordinates against the extrapolated coordinates and omit thetransmission of specified packets of data if the extrapolatedcoordinates and the real coordinates are substantially similar.

In some embodiments, the multi-dimensional pointing device 102 includesa plurality of buttons. The plurality of buttons allows users thatprefer a conventional user interface (e.g., arrow keys, etc.) tocontinue using the conventional user interface. In these embodiments,the host device drivers 314 may need to interpret a combination of thesebuttons as a single event to be conveyed to the middleware 313 of thehost system.

In some embodiments, the host device drivers 314 are configured so thatthe multi-dimensional pointing device 102 appears as a two-dimensionalpointing device (e.g., mouse, trackpad, trackball, etc.).

FIG. 4 is a block diagram illustrating inputs, outputs, and operationsof an exemplary device-side firmware 400 for the multi-dimensionalpointing device 102, according to some embodiments. Sensors 401 generatemeasurements that may be sampled by one or more sampling circuits 402.

In some embodiments, the sampled sensor measurements are packetized fortransmission 407 and transmitted to the host system 101 by a transmitter408.

In some embodiments, the sensors 401 are calibrated and corrected 403.For example, the sensors 401 may be calibrated and corrected so that aKalman filter that is used to compute the attitude of amulti-dimensional pointing device (e.g., the multi-dimensional pointingdevice 102 in FIG. 1, etc.) is initialized with a zero assumed error.The Kalman filter states are then determined 404. The determined Kalmanfilter states are then mapped to physical coordinates 405, and datarepresenting the physical coordinates are packetized for transmission407 by the transmitter 408. Keypad and other inputs 406 may also bepacketized for transmission 407 and transmitted by the transmitter 408.In some embodiments, the keypad and/or other inputs 406 are used inconjunction movements of the multi-dimensional pointing device 102 toproduce gestures that convey commands to a host system. In some of theseembodiments, the keypad and other inputs 406 are mapped to physicalcoordinates 405 (e.g., noting the physical coordinates at which thekeypad and other inputs were activated) prior to being packetized fortransmission 407. Alternately, the time ordered sequence in which keypadpresses (or other inputs) and changes in position of themulti-dimensional pointing device 102 are packetized and transmitted tothe host system is used by the device to determine the context of thekeypad presses (or other inputs) and to determine what gesture(s) wereperformed by the user.

The measurements from the sensors and the determined change in positionand/or attitude may also be used to enter and/or exit sleep andwake-on-movement modes 409.

In some embodiments, the multi-dimensional pointing device 102 measuresrotations of the remote over a physical space that is independent of thesize, distance and direction of the display of the host system 101. Infact, the multi-dimensional pointing device 102 may report onlydisplacements between two consecutive samples in time. Thus, theorientation of the multi-dimensional pointing device 102 does notmatter. For example, yaw may be mapped to left/right cursor movement andpitch may be mapped to up/down cursor movements.

In some embodiments, to conserve system power, the multi-dimensionalpointing device 102 detects a lack of movement of the multi-dimensionalpointing device 102 and puts itself into a low power (e.g., sleep) mode.In some embodiments, a single accelerometer is used to sense whether themulti-dimensional pointing device 102 is being moved and to generate aninterrupt to wake (e.g., wake-on-demand) the multi-dimensional pointingdevice 102 from the sleep mode.

In some embodiments, the multi-dimensional pointing device 102determines that it should enter a sleep mode based on one or more of thefollowing conditions: the magnitude of the acceleration measurement(e.g., A_(observed)) is not greater or smaller than the magnitude ofEarth's gravity (e.g., G) by a specified threshold, the standarddeviation of A_(observed) does not exceed a specified threshold, and/orthere is an absence of change in the angular relationship between themeasurement of the Earth's magnetic field (e.g., B) and A_(observed)greater than a specified threshold. Each of the aforementionedconditions may be used to indicate that the multi-dimensional pointingdevice 102 has entered a resting state (e.g., no substantial movement).After the multi-dimensional pointing device 102 has remained in aresting state for a specified number of consecutive samples, themulti-dimensional pointing device 102 enters a sleep mode.

In some embodiments, the device-side firmware 400 of themulti-dimensional pointing device 102 is updated by the host system 101via a wireless interface.

Some embodiments provide one or more games and/or demo applications thatdemonstrate how to use the multi-dimensional pointing device (e.g.,movement, controlling objects in the user interface, gestures, etc.).

Calculating Attitude During Dynamic Acceleration

FIG. 5 is a diagram 500 illustrating exemplary gravity (G) and magneticfield (B) vectors that can be used to determine attitude, according tosome embodiments. In some embodiments, G and B correspond to the Earth'sgravity and the Earth's magnetic field, respectively. The Earth'smagnetic field and gravity are assumed to form two stationary vectors.Using a magnetometer and an accelerometer, B and G may be measured. Forexample, the magnetic field vector B 501 and acceleration vector G 502may be measured. When the multi-dimensional pointing device 102 isrotated, and then held stationary, B and G are measured again. Inparticular, the magnetic field vector B 503 and the acceleration vectorG 504 may be measured. Given an unchanging relationship between B and G,the rotational operation that rotates B 501 and G 502 to B 503 and G504, respectively, can be calculated. This rotation operation is therelative attitude/heading change.

Before continuing with the discussion, it is instructive to define twoterms: body frame and the Earth frame. The body frame is the coordinatesystem in which B and G are measured with respect to a fixed point onthe multi-dimensional pointing device 102. The diagram 500 in FIG. 5illustrates the effect of a rotation of the multi-dimensional pointingdevice 102 as observed from the body frame. As the multi-dimensionalpointing device 102 is held with one end or point of themulti-dimensional pointing device 102 at a fixed position, rotation ofthe multi-dimensional pointing device 102 causes B and G to move withrespect to the body frame.

The Earth frame is the coordinate system in which B and G are measuredwith respect to a fixed point on the surface of the Earth. The Earthframe is typically the frame of reference for the user 103 of themulti-dimensional pointing device 102. When the user 103 moves themulti-dimensional pointing device 102, the user 103 typically thinksabout the motion relative to the Earth frame.

Thus, the solution to the attitude of the multi-dimensional pointingdevice 102 can be formulated as follows: given two measurements of twoconstant vectors taken with respect to a body frame (of themulti-dimensional pointing device 102) that has undergone a rotation,solve for the rotation of the multi-dimensional pointing device 102 inthe Earth frame.

There are a number of techniques can determine the attitude of themulti-dimensional pointing device 102. As discussed above, TRIAD is onesuch technique. Note that the following calculations may be formulatedusing Quaternion-based arithmetic to avoid issues with singularityassociated with the TRIAD technique. The TRIAD technique operates asfollows.

Given w₁ and w₂, which represent measurements (observations) of the Band G vectors in the body frame, the following are defined:

$\begin{matrix}{r_{1} = \frac{w_{1}}{w_{1}}} & (1) \\{r_{2} = \frac{r_{1} \times w_{2}}{{r_{1} \times w_{2}}}} & (2) \\{r_{3} = {r_{1} \times r_{2}}} & (3)\end{matrix}$

where, r₁ is the normalized column vector w₁, r₂ is a normalized columnvector orthogonal to r₁ and w₂, and r₃ is a normalized column vectororthogonal to r₁ and r₂.

Correspondingly, B and G are also known in the Earth frame. Howeverthese measurements are known a-priori; that is, they do not need to bemeasured and may be calculated from well-known theoretical models of theearth. For example, the magnitude and direction of the earth's magneticand gravitational fields in San Jose, Calif. can be calculated withoutmaking new measurements. Thus the measurements in the body frame may becompared relative to these known vectors. If we call the vectorsrepresenting B and G in the Earth frame v₁ and v₂, then we may define:

$\begin{matrix}{s_{1} = \frac{v_{1}}{v_{1}}} & (4) \\{s_{2} = \frac{s_{1} \times v_{2}}{{s_{1} \times v_{2}}}} & (5) \\{s_{3} = {s_{1} \times s_{2}}} & (6)\end{matrix}$

where s₁ is the normalized column vector v₁, s₂ is a normalized columnvector orthogonal to s₁ and v₂, and s₃ is a normalized column vectororthogonal to s₁ and s₂.

Using the normalized column vectors defined above, the attitude matrix(A) that gives the rotational transform (e.g., for generating anuncorrected attitude of the multi-dimensional pointing device 200) inthe Earth frame is:

A=R·S ^(T)  (7)

where R=[r₁|r₂|r₃] (e.g., a matrix comprised of the three column vectorsr₁, r₂, and r₃), S=[s₁|s₂|s₃] (e.g., a matrix comprised of the threecolumn vectors s₁, s₂, and s₃), and the “T” superscript denotes thetranspose of the matrix to which it is applied.

Applying to the problem at hand, if v₁ and v₂ are given as the B and Gvectors in the Earth frame and w₁ and w₂ are inferred from measurementsproduced by the multi-dimensional accelerometers 201-202 and themulti-dimensional magnetometer 203, the TRIAD technique may be used tocompute the uncorrected attitude A of the multi-dimensional pointingdevice 102.

As discussed above, the accuracy of the relative heading/attitude of themulti-dimensional pointing device 102 determined by the TRIAD techniqueis predicated on the assumption that the device is not subject todynamic acceleration. This assumption does not hold true inmulti-dimensional pointing applications, in which the user 103 makescontinuous movements and/or gestures with the multi-dimensional pointingdevice 102. FIG. 6 is a diagram 600 illustrating an attitudedetermination error caused at least in part by dynamic acceleration. Att=0, an acceleration measurement A_(OBS) 602 (i.e., Earth's gravity G)and a magnetic field measurement B 601 are measured. As themulti-dimensional pointing device 102 is rotated at t=1, an accelerationA_(DYN) 606 is induced on the multi-dimensional pointing device 102 sothat the vector combination of Earth's gravity G 605 and A_(DYN) 606produce an acceleration measurement A_(OBS) 604 in the body frame. Thus,the acceleration measurement A_(OBS) 604 does not measure G 605.Instead, it includes the error induced by A_(DYN) 606. Note that amagnetic field measurement B 603 is also measured in the body frame att=1. Accordingly, an attitude calculation using A_(OBS) 604 and B 603would include error due to the dynamic acceleration. Thus, the TRIADtechnique introduces an error to the computed attitude proportionate tothe size of A_(DYN) 606.

In order to solve the aforementioned problems, some embodiments includetwo or more accelerometers to measure the dynamic acceleration that themulti-dimensional pointing device 102 experiences. FIG. 7 is a diagram700 illustrating an exemplary technique for compensating for dynamicacceleration in attitude calculations of a multi-dimensional pointingdevice 701, according to some embodiments. The multi-dimensionalpointing device 701 may be any one of the multi-dimensional pointingdevices 102 and 200 in FIGS. 1 and 2, respectively. Themulti-dimensional pointing device 701 includes multi-dimensionalaccelerometers 703 (A) and 704 (B) separated by a distance D 710.Furthermore, the distance from a pivot origin 702 to themulti-dimensional accelerometer 703 (A) is equal to r_(rot) 720. Thepivot origin 702 may be offset from the axis formed by themulti-dimensional accelerometers 703 (A) and 704 (B) by a distance L722. For example, the distance L 722 may represent the offset betweenthe axis of the multi-dimensional accelerometers 703 (A) and 704 (B) anda wrist of the user 103 as the multi-dimensional pointing device 701 isheld in the hand of the user 103.

Dynamic acceleration experienced the multi-dimensional pointing device701 may include translational acceleration imparted by lateral movementof the multi-dimensional pointing device 701 and rotationalacceleration. When the multi-dimensional pointing device 701 is affectedby translational acceleration, both multi-dimensional accelerometers703-704 experience the same dynamic acceleration. When the device isaffected by angular acceleration, the multi-dimensional accelerometers703-704 experience dynamic acceleration proportional to their distancefrom the pivot origin 702.

For example, consider the case when the multi-dimensional pointingdevice 701 is pivoted about the pivot origin 702, causing themulti-dimensional accelerometers 703 and 704 to produce compositeacceleration measurements A_(OBS) 705 and A_(OBS) 706. The compositeacceleration measurement A_(OBS) 705 is a vector sum of the accelerationcaused by Earth's gravity (G 707) and the dynamic acceleration aexperienced by the first multi-dimensional accelerometer 703 (A). Thecomposite acceleration measurement A_(OBS) 706 is a vector sum of theacceleration caused by Earth's gravity (G 707) and the dynamicacceleration b experienced by the second multi-dimensional accelerometer704 (B). Note that since the multi-dimensional accelerometer 704 isfarther from the pivot origin 702 than the multi-dimensionalaccelerometer 703, the acceleration due to the rotation about the pivotorigin 702 is greater at the second multi-dimensional accelerometer 704(B) than at the first multi-dimensional accelerometer 703 (A). A_(OBS)705 and A_(OBS) 706 include errors 708 and 709, respectively.

The change in the attitude of the multi-dimensional pointing device 102may be computed using measurements from both of the twomulti-dimensional accelerometers 703-704. When the dynamic accelerationis entirely translational, the difference between the two computedattitudes is zero. In some embodiments, only rotational movement istranslated into cursor movements. Thus, translational displacements donot result in translational cursor movement because purely translationalmovements do not affect yaw, pitch or roll. However, when the dynamicacceleration includes rotational components, the difference between thetwo accelerometer measurements produced by the two multi-dimensionalaccelerometers 703-704 is used to substantially reduce the error in thecalculated attitude of the multi-dimensional pointing device 701 that iscaused by dynamic acceleration.

Determining Attitude Using a Kalman Filter

In some embodiments, the attitude of a multi-dimensional pointing device(e.g., the multi-dimensional pointing device 102 in FIG. 1, etc.) isdetermined by using a Kalman filter. Specifically, the Kalman filter maybe an extended Kalman filter. Note that this specification uses the term“Kalman filter” to refer to an “extended Kalman filter”.

Attention is now directed to FIG. 8, which is a block diagramillustrating an exemplary method 800 for determining an attitude of adevice undergoing dynamic acceleration, according to some embodiments.The Kalman filter generally includes two phases: a “predict” phase andan “update” phase. In the predict phase (802), an estimated state of theKalman filter (which can also be considered to be a state of the device)from the previous timestep is used to produce a predicted estimate ofthe state (e.g., a “predicted state”) at a current timestep. Timestepsare sometimes called epochs, or update periods. In the update phase(806), measurements (e.g., the acceleration measurements 204-205, themagnetic field measurement 206, etc.) sampled (804) from the sensors ofthe multi-dimensional pointing device (e.g., the multi-dimensionalaccelerometers 201-202, the multi-dimensional magnetometer 203, etc.)are used to correct the predicted state at the current timestep toproduce an “updated state” (e.g., the estimated state that is used inthe next time step). A mapping (808) is applied to the body rotationrate ω (e.g., obtained from the state vector of the Kalman filter) toconvert (810) ω into the cursor motion. After determining the attitudeof the multi-dimensional pointing device, the method then returns to the“predict phase” (802) at the next timestep. In some embodiments, therepeat rate of the method ranges from as slow as twenty times per secondto as high as about 200 times per second, corresponding to timestepsranging from as large as 50 milliseconds to as small as about 5millisecond.

In some embodiments, during the predict phase, a predicted state{circumflex over (x)} and a predicted error covariance matrix P aredetermined as follows:

$\begin{matrix}{{\hat{x}\left( t_{k + 1} \right)} = {\int\limits_{t_{k}}^{t_{k + 1}}{{f\left( {x,u,t} \right)}{t}}}} & (8) \\{{P_{k}\left( t_{k + 1} \right)} = {{\Phi \left\lbrack {{P_{k}\left( t_{k} \right)} + {Q\left( t_{k} \right)}} \right\rbrack}\Phi^{- 1}}} & (9)\end{matrix}$

where {circumflex over (x)}(t_(k+1)) is the predicted state of theKalman filter at timestep k+1, ƒ(x,u,t) are the dynamics of the system(defined below), x is the state, u is a control input (e.g.,accelerations due to the arm of the user 103), t is time, P_(k)(t_(k))is the predicted error covariance matrix at timestep k, P_(k)(t_(k+1))is the predicted error covariance matrix at timestep k+1, Q(t_(k)) is anapproximation of the process noise matrix at timestep k, and Φ is astate transition matrix, which is obtained from the system dynamics.

The state transition matrix, Φ, is nominally an identity matrix (i.e.,ones on the diagonal) for those states that do not have a dynamicsmodel. A dynamics model is a model of the underlying dynamic system. Forexample, the dynamics model for a body in motion may include Newton'sequations of motion. In some embodiments, the dynamics model forattitude determination is defined by Equations (15)-(21) below. In someembodiments, only the quaternion representing the attitude of themulti-dimensional pointing device and the vector including valuesrepresenting the body rotation rate are associated with dynamic models.Thus, the only non-zero off-diagonal elements of the state transitionmatrix Φ are the portions of the state transition matrix that correspondto the covariances of the quaternion and body rotation rate states.Numerical values for this portion of the state transition matrix may becalculated for each timestep using a finite difference scheme instead ofcalculation of the dynamic system's Jacobian matrix. (Note that findingand integrating the Jacobian is the traditional technique of computingthe state transition matrix.) In this finite difference scheme, a set ofperturbed state vectors at time t_(k), as well as the unperturbed state,are propagated through the dynamics model (e.g., represented byequations (15)-(21) below). Each perturbed state vector is perturbed ina single state. The differences between the propagated perturbed stateand the propagated unperturbed state are calculated. The differencevectors are divided by size of the initial perturbation. Thesedifference vectors make up the dynamic portion of the state transitionmatrix.

In some embodiments, the process noise matrix, Q, only includes valueson the diagonal elements of the matrix.

In some embodiments, the state of the Kalman filter includes a statevector defined as follows:

$\begin{matrix}{\hat{x} = \begin{bmatrix}\overset{\rightarrow}{q} \\\overset{\rightarrow}{\omega} \\r_{rot} \\a_{Yd} \\a_{Zd}\end{bmatrix}} & (10)\end{matrix}$

where {right arrow over (q)} is a vector including values of aquaternion representing the attitude of the multi-dimensional pointingdevice, {right arrow over (ω)} is a vector including values representingthe body rotation rate (e.g., the rate at which the attitude of themulti-dimensional pointing device is rotating), r_(rot) is a vectorincluding a value that represents the radius of rotation between one ofthe multi-dimensional accelerometers (e.g., the multi-dimensionalaccelerometer 703 (A)) and the pivot origin (e.g., the pivot origin702), a_(Yd) and a_(Zd) are the bias values in the Y and Z directions ofthe difference between the two accelerometer measurements (e.g., theaccelerometer measurements 204-205). In some embodiments, the bias ofthe multi-dimensional magnetometer is estimated using a separate Kalmanfilter.

Before continuing with the discussion of the Kalman filter, it isinstructive to discuss the quaternion {right arrow over (q)}representing the attitude of the multi-dimensional pointing device. FIG.9 is a graph illustrating an exemplary quaternion 900, according to someembodiments. Any rotation (e.g., from one frame of reference to another,or from one attitude of a device to another) may be represented by athree-dimensional unit vector {circumflex over (n)} having componentsn_(x), n_(y), and n₂, and an angle θ, which is the rotation about theunit vector {circumflex over (n)}. The rotation may be expressed as anormalized four-dimensional quaternion {right arrow over (q)} having thecomponents q₁, q₂, q₃, and q₄ as follows:

$\begin{matrix}{q_{1} = {n_{x}\sin \frac{\theta}{2}}} & (11) \\{q_{2} = {n_{y}\sin \frac{\theta}{2}}} & (12) \\{q_{3} = {n_{z}\sin \frac{\theta}{2}}} & (13) \\{q_{4} = {\cos \frac{\theta}{2}}} & (14)\end{matrix}$

Returning to the discussion of the Kalman filter, in some embodiments,the function ƒ(x,u,t) represents the equations of motion. For example,the equations of motion may be:

$\begin{matrix}{\overset{.}{\overset{\rightarrow}{q}} = {\left\lbrack \overset{\sim}{\omega} \right\rbrack \overset{\rightarrow}{q}}} & (15) \\{\overset{.}{\overset{\rightarrow}{\omega}} = {h\left( {{\overset{\rightarrow}{a}}_{diff},\overset{\rightarrow}{\omega}} \right)}} & (16) \\{\left\lbrack \overset{\sim}{\omega} \right\rbrack = {\frac{1}{2}\begin{bmatrix}0 & {- \omega_{x}} & {- \omega_{y}} & {- \omega_{z}} \\\omega_{x} & 0 & \omega_{z} & {- \omega_{y}} \\\omega_{y} & {- \omega_{z}} & 0 & \omega_{x} \\\omega_{z} & \omega_{y} & \omega_{x} & 0\end{bmatrix}}} & (17)\end{matrix}$

where {right arrow over ({dot over (q)} is the first time derivative ofthe quaternion {right arrow over (q)} representing the attitude of themulti-dimensional pointing device, {tilde over (ω)} (e.g., see Equation(17), where the components ω_(x), ω_(y), and ω_(z) are the x, y, and zcomponents of {right arrow over (ω)}) is the linear mapping of the bodyrates that when multiplied by quaternion state yields the time rate ofchange of the quaternion state, {right arrow over ({dot over (ω)} is theangular acceleration (e.g., first time derivative of the body rotationrate) of the multi-dimensional pointing device, h({right arrow over(a)}_(diff), {right arrow over (ω)}) is a function of the vectorrepresenting the difference between the two accelerometer measurements({right arrow over (a)}_(diff)) and the body rotation rate vector({right arrow over (ω)}). h({right arrow over (a)}_(diff), {right arrowover (ω)}) is defined below.

Each multi-dimensional accelerometer measures a composite (e.g., vectorsum) of the following accelerations/forces: tangential, centripetal,gravitational (as measured in the body frame of the accelerometer), andtranslational. These acceleration components may be represented asfollows:

{right arrow over (a)} _(A) =−{right arrow over ({dot over (ω)}×{rightarrow over (r)} _(A) −{right arrow over (ω)}×{right arrow over(ω)}×{right arrow over (r)} _(A)+DCM({right arrow over (q)}){right arrowover (g)}+{right arrow over (a)} _(translational)  (18)

{right arrow over (a)} _(B) =−{right arrow over ({dot over (ω)}×{rightarrow over (r)} _(B) −{right arrow over (ω)}×{right arrow over(ω)}×{right arrow over (r)} _(B)+DCM({right arrow over (q)}){right arrowover (g)}+{right arrow over (a)} _(translational)  (19)

where {right arrow over (a)}_(A) and {right arrow over (a)}_(B) are thecomposite accelerations measurements (e.g., the accelerationmeasurements 204-205) for each of the two accelerometers (e.g., themulti-dimensional accelerometers 201-202) of the multi-dimensionalpointing device, {right arrow over ({dot over (ω)} is the rate of changeof the body rotation rate {right arrow over (ω)}, {right arrow over(r)}_(A) and {right arrow over (r)}_(B) are the radius of rotations ofeach of the two accelerometers relative to a pivot origin, DCM({rightarrow over (q)}) is the direction cosine matrix (DCM) that is obtainedfrom the quaternion {right arrow over (q)} representing the attitude ofthe multi-dimensional pointing device (e.g., the {right arrow over (q)}is converted to a DCM so that it can operate on the gravity vector{right arrow over (g)}), {right arrow over (g)} is the acceleration dueto gravity as viewed from the body frame (e.g., the frame of theaccelerometer), and {right arrow over (a)}_(translational) is thetranslational acceleration.

Note that the Kalman state described above only includes a state valuerepresenting the radius of rotation, r_(rot), to one of theaccelerometers (e.g., the multi-dimensional accelerometer 703 (A)). Ifthe offset (e.g., L 722, FIG. 7) between the pivot origin (e.g., thepivot origin 702) and the axis of the accelerometers (e.g., themulti-dimensional accelerometers 703-704) are collinear (e.g., L 722 iszero), the magnitude of {right arrow over (r)}_(B) is r_(rot) (e.g.,r_(rot) 720) plus the distance between the accelerometers (e.g., D 710,which is a known quantity). If the offset between the pivot origin andthe axis of the accelerometers is non-zero, {right arrow over (r)}_(B)may be calculated from the geometric relationship between, {right arrowover (r)}_(A), D 710, r_(rot), and the offset (e.g., by using thePythagorean Theorem, etc.), where r_(rot) and the offset are states ofthe Kalman filter.

A vector difference {right arrow over (a)}_(diff) between {right arrowover (a)}_(A) and {right arrow over (a)}_(B) yields:

{right arrow over (a)} _(diff) ={right arrow over (a)} _(B) −{rightarrow over (a)} _(A) =−{right arrow over ({dot over (ω)}×{right arrowover (r)} _(diff) −{right arrow over (ω)}×{right arrow over (ω)}×{rightarrow over (r)} _(diff)  (20)

where, {right arrow over (r)}_(diff) is the vector difference between{right arrow over (r)}_(A) and {right arrow over (r)}_(B) (e.g., {rightarrow over (r)}_(diff)={right arrow over (r)}_(B)−{right arrow over(r)}_(A)). Note that {right arrow over (a)}_(diff) does not include theacceleration forces due to gravity and translation.

Equation (20) may be rearranged to solve for the angular acceleration{right arrow over ({dot over (ω)}:

$\begin{matrix}{{\overset{.}{\overset{\rightarrow}{\omega}}_{\overset{.}{\overset{\rightarrow}{\omega}} \cdot {\overset{\rightarrow}{r}}_{diff}}} = {{\frac{1}{{{\overset{\rightarrow}{r}}_{diff}}^{2}}\left\lbrack {{\overset{\rightarrow}{a}}_{diff} + {\overset{\rightarrow}{\omega} \times \overset{\rightarrow}{\omega} \times {\overset{\rightarrow}{r}}_{diff}}} \right\rbrack} \times {\overset{\rightarrow}{r}}_{diff}}} & (21)\end{matrix}$

where {right arrow over ({dot over (ω)} is evaluated at {right arrowover ({dot over (ω)}·{right arrow over (r)}_(diff)=0 (e.g., when theonly non-zero components of the angular acceleration {right arrow over({dot over (ω)}, are orthogonal to the vector {right arrow over(r)}_(diff), which is defined in paragraph [00100]). Equation (21) isthen used in Equation (16). Note that a_(diff) is a measurement (e.g.,from the multi-dimensional accelerometers), ω is obtained from statevector, and {right arrow over (r)}_(diff) is the vector differencebetween {right arrow over (r)}_(A) and {right arrow over (r)}_(B), asexplained above.

In some embodiments, the number of states in the error covariance matrixP is reduced by expressing the variation of the quaternion state asorthogonal modified Rodrigues parameters (MRPs), which have three (3)parameters as compared to four (4) parameters in a quaternion. The MRPand the quaternion contain the same rotation information, but theredundant parameter in the quaternion avoids singularities. In theseembodiments, the update of the quaternion state is estimated as an MRProtation, and then converted to a quaternion. The update of thequaternion state is applied multiplicatively and preserves the unit normproperty of the quaternion.

During the update phase, the predicted state matrix and predicted errorcovariance matrix are updated based on the sensor measurement asfollows:

{circumflex over (x)} _(k+1)(t _(k))={circumflex over (x)}(t _(k+1))+K_(k)({right arrow over (y)} _(m) −ŷ)  (22)

P _(k+1)(t _(k))=(I−K _(k) G _(k))P _(k)(t _(k))  (23)

where {circumflex over (x)}_(k+1)(t_(k)) is the updated state vector attimestep k+1, {circumflex over (x)}(t_(k+1)) is the predicted statevector at timestep k that was calculated in the predict phase, K_(k) isthe Kalman gain, {right arrow over (y)}_(m) is the observed measurements(e.g., the sensor measurements), ŷ is the predicted sensor measurements(e.g., the predicted sensor measurements that are obtained from thepredicted state vector and the sensor models described in equations (28)and (29) below), I is the identity matrix, and G_(k) is an observationtransformation matrix that maps the deviations from the state vector todeviations from the observed measurements (e.g., the sensormeasurements). Note that the term {right arrow over (y)}_(m)−ŷ isreferred to as a residual.

Generally, ŷ is a function of the state vector, the first timederivative of the state vector, and time (e.g., ŷ=g({right arrow over(x)},{right arrow over ({dot over (x)},t)), and may be determined usingthe sensor models described below. The Kalman gain K_(k) may bedetermined using the following equations:

$\begin{matrix}{K_{k} = {P_{k}G_{k}^{T}S^{- 1}}} & (24) \\{S_{k} = {{G_{k}P_{k}G_{k}^{T}} + R}} & (25) \\{G_{k} = \frac{\partial\hat{y}}{\partial\overset{\rightarrow}{X}}} & (26)\end{matrix}$

where R is the measurement covariance matrix.

In some embodiments, {right arrow over (y)}_(m) includes the followingcomponents:

$\begin{matrix}{{\overset{\rightarrow}{y}}_{m} = \begin{bmatrix}{\overset{\rightarrow}{H}}_{xy} \\{\overset{\rightarrow}{a}}_{A} \\{\overset{\rightarrow}{a}}_{B}\end{bmatrix}} & (27)\end{matrix}$

where {right arrow over (H)}_(xy) is the directional residual of themagnetic field measurement (e.g., the magnetic field measurement 206),{right arrow over (a)}_(A) is the accelerometer measurement (e.g., theaccelerometer measurement 205) from a first multi-dimensionalaccelerometer (e.g., the multi-dimensional accelerometer 202), and{right arrow over (a)}_(B) is the accelerometer measurement (e.g., theaccelerometer measurement 204) from a second multi-dimensionalaccelerometer (e.g., the multi-dimensional accelerometer 201). Note thatthe directional residual of the magnetic field measurement, {right arrowover (H)}_(xy), may be used because when determining the attitude of amulti-dimensional pointing device, only the directional information isrequired; the magnitude of the magnetic field is not used. In fact, inthese embodiments, attempting to correct/update the magnitude of themagnetic field in the Kalman filter state causes the Kalman filter stateto diverge. {right arrow over (H)}_(xy) may be calculated from themagnetic field measurement using the technique described in “SpinningSpacecraft Attitude Estimation Using Markley Variables: FilterImplementation and Results” (Joseph E. Sedlak, 2005, available athttp://www.ai-solutions.com/library/tech.asp), which is herebyincorporated by reference in its entirety.

In some embodiments, the sensor model for the multi-dimensionalmagnetometer and the multi-dimensional accelerometers are:

Ĥ _(xy) =[R _(Bzenith)][DCM({circumflex over (q)}(t _(k+1)))]{rightarrow over (H)} _(ref)  (28)

â=−{right arrow over ({dot over (ω)}×{right arrow over (r)}_(Acc)−{circumflex over (ω)}(t _(k+1))×{circumflex over (ω)}(t_(k+1))×{right arrow over (r)} _(Acc)+DCM({right arrow over (q)}(t_(k+1))){right arrow over (g)}  (29)

where Ĥ_(xy) is the two-dimensional directional residual between themeasured and estimated magnetometer values, R_(Bzenith) is a rotationmatrix that rotates the magnetic field measurement to the Z-axis vectorin the new frame of reference (e.g., the frame of reference described in“Spinning Spacecraft Attitude Estimation Using Markley Variables: FilterImplementation and Results,” whereby the directional variances of athree dimensional vector are expressed as two variables.),DCM({circumflex over (q)}(t_(k+1))) is the DCM that is obtained from thequaternion {circumflex over (q)} representing the estimated attitude ofthe multi-dimensional pointing device (e.g., the {circumflex over (q)}is converted to a DCM so that it can operate on the gravity vector{right arrow over (g)} and/or {right arrow over (H)}_(ref)), {rightarrow over (H)}_(ref) is the assumed magnetic field measurement in theEarth frame, and {right arrow over (r)}_(Acc) is the radius of rotationfor a respective accelerometer, relative to the pivot point. The angularacceleration {right arrow over ({dot over (ω)} may be obtained from thedifference of the accelerometer measurements (e.g., Equation (21)) andacts as a “pass-through” variable for the sensor measurements

In some embodiments, the state vector {circumflex over (x)} is a 10×1matrix, the error covariance matrix P is a 9×9 matrix, and theobservation partial derivative matrix G is an 8×9 matrix. In theseembodiments, {right arrow over (q)} is a 4×1 vector, {right arrow over(ω)} a 3×1 vector, r_(rot) is a 1×1 vector, and a_(Yd) and a_(Zd) areeach 1×1 vectors. These components of the state vector {circumflex over(x)} together form a 10×1 matrix.

Accelerometer quantization may cause the attitude determined by theKalman filter to incorrectly indicate that the multi-dimensionalpointing device is moving when it is not. If left uncorrected,accelerometer quantization may significantly degrade performance of thesystem in which the multi-dimensional pointing device is used (e.g., thecursor on the host system may drift across the user interface). Thus, insome embodiments, for small values of the accelerometer measurements(e.g., values below twenty times the quantization interval), thetechniques described in “Covariance Profiling for an Adaptive KalmanFilter to Suppress Sensor Quantization Effects” by D. Luong-Van et al.(43rd IEEE Conference on Decision and Control, Volume 3, pp. 2680-2685,14-17 Dec. 2004), which is hereby incorporated by reference in itsentirety, are used to mitigate the effects of the quantized datameasurements reported by the accelerometers.

Furthermore, accelerometer noise may cause jitter causing the attitudedetermined by the Kalman filter to indicate that the multi-dimensionalpointing device is moving even when the multi-dimensional pointingdevice at rest. Thus, in some embodiments, a deadband is used for valuesof the accelerometer measurements that occur in a specified range ofquantization levels of the accelerometer measurements. For example, thespecified range may be between two and twenty times the quantizationlevel of the accelerometers. Note that it is desirable to minimize thedeadband, but this minimization must be balanced against the deviceperformance at low angular rates and accelerations where quantizationeffects will dominate the behavior of the pointer.

Adaptive Kalman Gain

As discussed above, substantial error can arise in the calculation ofthe attitude of a multi-dimensional pointing device that is undergoingdynamic acceleration. These errors arise from the inability of a singlemulti-dimensional accelerometer to distinguish between the effects ofdynamic acceleration and the actual gravity vector. To compensate forthis, in some embodiments, the acceleration measurements from theaccelerometers are given less weight when the multi-dimensional pointingdevice is undergoing dynamic acceleration than when themulti-dimensional pointing device is not undergoing dynamicacceleration.

The weight of the acceleration measurements in the Kalman filter may becontrolled by the Kalman gain (K_(k)). Thus, in some embodiments, theKalman gain is adjusted based on the amount of dynamic accelerationexperienced by the multi-dimensional pointing device. For example, theKalman gain may be adjusted through the measurement covariance matrix R(see equations 24 and 25, above).

Attention is now directed to FIG. 10, which is a flow diagram of amethod 1000 for determining an attitude of a device undergoing dynamicacceleration, according to some embodiments. A difference between afirst accelerometer measurement received from a first multi-dimensionalaccelerometer of the device and a second accelerometer measurementreceived from a second multi-dimensional accelerometer of the device iscalculated (1002) (e.g., see Equation (20)).

A Kalman gain based on the difference is adjusted (1004), wherein theKalman gain is used in a Kalman filter that determines the attitude ofthe device. When the difference is less than a specified threshold,values associated with the first accelerometer measurement and thesecond accelerometer measurement in a measurement covariance matrix ofthe Kalman filter (e.g., R) are decreased so that the firstaccelerometer measurement and the second accelerometer measurement aregiven more weight in the Kalman filter relative to the magnetic fieldmeasurement than when the difference is greater than the specifiedthreshold. When the difference is greater than a specified threshold,covariance values associated with the first accelerometer measurementand the second accelerometer measurement in a measurement covariancematrix of the Kalman filter (e.g., R) are increased so that the firstaccelerometer measurement and the second accelerometer measurement aregiven less weight in the Kalman filter relative to the magnetic fieldmeasurement than when the difference is less than the specifiedthreshold. For example, when the difference is greater than thespecified threshold, the covariance values associated with the firstaccelerometer measurement and the second accelerometer measurement maybe increased by a factor of 100 compared with their values when thedifference is less than the specified threshold. This threshold may bedefined as being the same differential acceleration threshold as definedfor the deadband.

An attitude of the device is determined (1006) using the Kalman filterbased at least in part on the Kalman gain, the first accelerometermeasurement, the second accelerometer measurement, and a magnetic fieldmeasurement received from a multi-dimensional magnetometer of thedevice. For example, the Kalman filter described above with reference toFIG. 8 and Equations (8)-(29) may be used to determine the attitude ofthe device.

FIG. 11 is a block diagram of a multi-dimensional pointing device 1100.The multi-dimensional pointing device 1100 may be any one of themulti-dimensional pointing devices 102, 200, and 701. Themulti-dimensional pointing device 1100 typically includes one or moreprocessing units (CPU's) 1102, one or more network or othercommunications interfaces 1104 (e.g., the transmitter 408, FIG. 4, orother wireless communication interface, as described above withreference to FIGS. 1 and 4), memory 1110, accelerometers 1170, and oneor more communication buses 1109 for interconnecting these components.In some embodiments, communications interfaces 1104 include atransmitter 408 (FIG. 4) for transmitting information, such asaccelerometer and magnetometer measurements, and/or the computedattitude of the multi-dimensional pointing device 1100, and/or otherinformation to a host system (e.g., the host system 101 or 1200). Thecommunication buses 1109 may include circuitry (sometimes called achipset) that interconnects and controls communications between systemcomponents. The multi-dimensional pointing device 1100 optionally mayinclude a user interface 1105 comprising a display device 1106 (LCDdisplay, LED display, etc.) and input devices 1107 (e.g., keypads,buttons, etc.). In some embodiments, the multi-dimensional pointingdevice 1100 includes one or more magnetometers 1172. Memory 1110includes high-speed random access memory, such as DRAM, SRAM, DDR RAM orother random access solid state memory devices; and may includenon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 1110 may optionallyinclude one or more storage devices remotely located from the CPU(s)1102. Memory 1110, or alternately the non-volatile memory device(s)within memory 1110, comprises a non-transitory computer readable storagemedium. In some embodiments, memory 1110 stores the following programs,modules and data structures, or a subset thereof:

-   -   an operating system 1112 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a communication module 1113 that is used for connecting the        multi-dimensional pointing device 1100 to a host system via the        one or more communication network interfaces 1104 (wired or        wireless); the communication module optionally may also be        adapted for connecting the multi-dimensional pointing device        1100 to one or more communication networks, such as the        Internet, other wide area networks, local area networks,        metropolitan area networks, and so on;    -   data representing accelerometer measurements 1114;    -   data representing magnetometer measurements 1115;    -   data representing button presses 1116;    -   a user interface module 1118 that receives commands from the        user via the input devices 1107 and generates user interface        objects in the display device 1106;    -   a gesture determination module 1119 that determines gestures        based on a sequence of corrected attitude measurements, as        described above; and    -   a Kalman filter module 1120 that determines the attitude of the        multi-dimensional pointing device 1100, as described above with        respect to FIGS. 8-10 and Equations (8)-(29), wherein the Kalman        filter module 1120 includes: a sensor model 1121 (e.g., the        sensor model described in Equations (28)-(29)), a dynamics model        1122 (e.g., the dynamics model described in Equations        (15)-(21)), a predict module 1123 that performs the predict        phase operations of the Kalman filter (e.g., step 802 in FIG.        8), an update module 1124 that performs the update operations of        the Kalman filter (e.g., step 806 in FIG. 8), a state vector        1125 of the Kalman filter (e.g., the state vector in Equation        (10)), a mapping 1126 (e.g., the mapping 808 in FIG. 8), Kalman        filter matrices 1127 (e.g., the Kalman filter matrices P, G, S,        K, R, etc., as described above), and attitude estimates 1128        (e.g., the attitude estimates as obtained from the quaternion in        the state vector {circumflex over (x)} in Equation (10)).

It is noted that in some of the embodiments described above, themulti-dimensional pointing device 1100 does not include a gesturedetermination module 1119, because gesture determination is performed bya host system. In some embodiments described above, themulti-dimensional pointing device 1100 also does not include the Kalmanfilter module 1120 because the multi-dimensional pointing device 1100transmits accelerometer and magnetometer measurements (and optionallybutton presses 1116) to a host system at which the attitude of thepointing device is determined.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and each of the aboveidentified programs or modules corresponds to a set of instructions forperforming a function described above. The set of instructions can beexecuted by one or more processors (e.g., the CPUs 1102). The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 1110 maystore a subset of the modules and data structures identified above.Furthermore, memory 1110 may store additional modules and datastructures not described above.

Although FIG. 11 shows a “multi-dimensional pointing device,” FIG. 11 isintended more as functional description of the various features whichmay be present in a pointing device. In practice, and as recognized bythose of ordinary skill in the art, items shown separately could becombined and some items could be separated.

FIG. 12 is a block diagram of a host system 1200. The host system 1200may be any one of the host systems 101, 300 described above. The hostsystem 1200 typically includes one or more processing units (CPU's)1202, one or more network or other communications interfaces 1204 (e.g.,any of the wireless interfaces described above with reference to FIG.1), memory 1210, and one or more communication buses 1209 forinterconnecting these components. In some embodiments, communicationsinterfaces 1204 include a receiver for receiving information, such asaccelerometer and magnetometer measurements, and/or the computedattitude of a multi-dimensional pointing device (e.g., multi-dimensionalpointing devices 102, 200, 400 or 1100), and/or other information fromthe multi-dimensional pointing device. The communication buses 1209 mayinclude circuitry (sometimes called a chipset) that interconnects andcontrols communications between system components. The host system 1200optionally may include a user interface 1205 comprising a display device1206 (LCD display, LED display, etc.) and input devices 1207 (e.g., amulti-dimensional pointing device, mouse, keyboard, trackpad, trackball,keypads, buttons, etc.). Memory 1210 includes high-speed random accessmemory, such as DRAM, SRAM, DDR RAM or other random access solid statememory devices; and may include non-volatile memory, such as one or moremagnetic disk storage devices, optical disk storage devices, flashmemory devices, or other non-volatile solid state storage devices.Memory 1210 may optionally include one or more storage devices remotelylocated from the CPU(s) 1202. Memory 1210, or alternately thenon-volatile memory device(s) within memory 1210, comprises anon-transitory computer readable storage medium. In some embodiments,memory 1210 stores the following programs, modules and data structures,or a subset thereof:

-   -   an operating system 1212 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks (e.g., the middleware 313 in FIG. 3);    -   a communication module 1213 that is used for connecting the host        system 1200 to a pointing device (e.g., the multi-dimensional        pointing device 1100), and/or other devices or systems via the        one or more communication network interfaces 1204 (wired or        wireless), and for connecting the host system 1200 to one or        more communication networks, such as the Internet, other wide        area networks, local area networks, metropolitan area networks,        and so on;    -   a user interface module 1214 that receives commands from the        user via the input devices 1207 and generates user interface        objects in the display device 1206;    -   a gesture determination module 1215 that determines gestures        based on a sequence of corrected attitude measurements for a        pointing device, as described above;    -   data representing an attitude estimate 1216 that is received        from a multi-dimensional pointing device;    -   data representing accelerometer measurements 1217 received from        a multi-dimensional positioning device and/or determined;    -   data representing magnetometer measurements 1218 received from a        multi-dimensional positioning device;    -   data representing button presses 1219 received from a        multi-dimensional positioning device; and    -   a Kalman filter module 1220 that determines the attitude of the        host system 1200, as described above with respect to FIGS. 8-10        and Equations (8)-(29), wherein the Kalman filter module 1220        includes: a sensor model 1221 (e.g., the sensor model described        in Equations (28)-(29)), a dynamics model 1222 (e.g., the        dynamics model described in Equations (15)-(21)), a predict        module 1223 that performs the predict phase operations of the        Kalman filter (e.g., step 802 in FIG. 8), an update module 1224        that performs the update operations of the Kalman filter (e.g.,        step 806 in FIG. 8), a state vector 1225 of the Kalman filter        (e.g., the state vector {circumflex over (x)} in Equation (10)),        a mapping 1226 (e.g., the mapping 808 in FIG. 8), Kalman filter        matrices 1227 (e.g., the Kalman filter matrices P, G, S, K, R,        etc., as described above), and attitude estimates 1228 (e.g.,        the attitude estimates as obtained from the quaternion in the        state vector {circumflex over (x)} in Equation (10)).

It is noted that in some of the embodiments described above, the hostsystem 1200 does not store data representing accelerometer measurements1217 and data representing magnetometer measurements 1218, and also doesnot include the Kalman filter module 1220 because the multi-dimensionalpointing device's accelerometer and magnetometer measurements areprocessed at the multi-dimensional pointing device, which sends datarepresenting the attitude estimate 1228 to the host system 1200. Inother embodiments, the multi-dimensional pointing device sends datarepresenting measurements to the host system 1200, in which case themodules for processing that data are present in the host system 1200.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and each of the aboveidentified programs or modules corresponds to a set of instructions forperforming a function described above. The set of instructions can beexecuted by one or more processors (e.g., the CPUs 1202). The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. The actual number of processors andsoftware modules used to implement the host system 1200 and how featuresare allocated among them will vary from one implementation to another.In some embodiments, memory 1210 may store a subset of the modules anddata structures identified above. Furthermore, memory 1210 may storeadditional modules and data structures not described above.

Rolling Gestures

As discussed above, existing pointing devices can be used to performtranslational movements of objects (e.g., a cursor, etc.) in a userinterface of a client computer system. One type of non-translationalmovement that can be performed in the real world (as opposed to thesimulated environment displayed by a computer) is a rolling gesture(e.g., a rotation operation). For example, the rolling gesture may beperformed when turning a key or turning a dial. However, existingpointing devices do not enable the performance of rolling gestures orrotation operations with respect to objects displayed in the userinterface of a computer system. Accordingly, it would be highlydesirable to provide a multidimensional pointing device (e.g., a freespace pointer) that sends commands to a computer system so as to provideboth translational and non-translational movements of objects in a userinterface of the computer system.

FIG. 13 presents a block diagram 1300 illustrating an exemplarymulti-dimensional pointing device 1302 performing a rolling gesture soas to control an object or control parameter in an exemplary userinterface 1308 of an exemplary host system 1306, according to someembodiments. The multi-dimensional pointing device 1302 may be any ofthe multi-dimensional pointing devices 102, 200, and 1100 discussedabove. The “host system” is any computer system or device that isconfigured to receive gesture data from the multi-dimensional pointingdevice 1302. The term “host system” is used interchangeably with theterm “client computer system” in this specification. The gesture datasent by the multi-dimensional pointing device 1302 to the host systemtypically includes translational movement data and rotational movementdata, and optionally includes button press data, and possibly includesother data as well.

In the example shown in FIG. 13, the user interface 1308 includes avolume control object 1310 that may be rotated by a user of themulti-dimensional pointing device 1302. The volume control object 1310can be any object or object element (e.g., a dial within a controlobject) that can be rotated. When the user rotates the multi-dimensionalpointing device 1302 about its longitudinal axis (i.e., a roll axis),the host system 1306 responds by rotating the volume control object 1310in the user interface 1308 in the same direction (e.g., clockwise orcounterclockwise) that the pointing device 1302 is being rotated.

In some other embodiments, the volume control object 1310 need notrotate, and instead a volume indicator within the volume control object1310 may move, change size or rotate as the multi-dimensional pointingdevice 1302 is rotated. For example, the volume control object 1310 caninclude a bar or wedge-shaped object that increases or decrease in sizeas the multi-dimensional pointing device 1302 is rotated.

In some embodiments, if there is more than one object in the userinterface 1308 that may be rotated or otherwise controlled by rotationof the multi-dimensional pointing device 1302, the object to be rotatedor controlled is selected using the multi-dimensional pointing device1302. There are several different ways that object selection can beaccomplished. In some embodiments, an object is selected by manipulatingthe multi-dimensional pointing device 1302 so as to position a cursor onor within a predetermined distance of the object, and then performing arotation gesture so as to control the object. Optionally, when the usermanipulates the multi-dimensional pointing device to move the cursorwithin the predetermined distance of the volume control object 1310, a“grab” operation is performed on the volume control object 1310.

Alternatively, an object is selected by manipulating themulti-dimensional pointing device 1302 so as position a cursor on orwithin a predetermined distance of the object and then pressing a buttonon the multi-dimensional pointing device 1302 so as to select theobject. In yet other embodiments, an object is selected by pressing acorresponding button on the multi-dimensional pointing device 1302. Forexample, the multi-dimensional pointing device 1302 may have two or morebuttons, each corresponding to a different respective user interfaceobject.

In other embodiments, other combinations of button presses and gesturesmay be used to perform the operations discussed with respect to FIG. 13.

In embodiments in which there is only one object in the user interface1308 that may be controlled by a rotation gesture performed using themulti-dimensional pointing device 1302, the user may press (andoptionally hold down) a gesture button while rotating themulti-dimensional pointing device 1302 about its longitudinal axis so asto control that object, without having to first select the object.

Once a user interface object has been selected, the selected object isrotated or otherwise manipulated by performing a rotation gesture usingthe multi-dimensional pointing device 1302. Performance of the rotationgesture results in rotation information (e.g., a rolling gesture metric)being transmitted to the host system 1306. More specifically, rollinggesture metric values may be conveyed periodically to the host system1306, each rolling gesture metric value representing either a change inrotational angle (of the multi-dimensional pointing device 1302) betweenconsecutive sample times, or a rate of rotational change (of themulti-dimensional pointing device 1302) between consecutive sampletimes. The rolling gesture metric values conveyed to the host system1306 are then used to manipulate (e.g., rotate or otherwise adjust) anobject on the user interface or a setting of the host system.

In some embodiments, when the user rotates the multi-dimensionalpointing device 1302 about its longitudinal axis, the host system 1306responds by rotating the volume control object 1310 in a directioncorresponding to the direction of rotation of the pointing device 1302,or by otherwise updating the volume control object 1310 in a mannerconsistent with the direction and amount of the rotation. In some ofthese embodiments, the user first presses and releases a volume buttonor gesture button on the multi-dimensional pointing device 1302 prior torotating the multi-dimensional pointing device 1302 about itslongitudinal axis, and then begins the rotation gesture within apredefined amount of time (e.g., 1 second). Alternatively, to performthe rotation gesture, the user presses and holds down a volume button orgesture button on the multi-dimensional pointing device 1302 whilerotating the multi-dimensional pointing device 1302 about itslongitudinal axis.

The discussion above refers to a volume control object 1310 in the userinterface 1308. However, the volume of a host system may be adjustedwithout requiring a volume control object 1310 in a user interface. Forexample, the user may press a volume button on the multi-dimensionalpointing device 1302 and then rotate the multi-dimensional pointingdevice 1302 to adjust the volume. In this example, the user receivesfeedback, in response to the rotation gesture, in the volume of thesounds output by the host system.

As noted above, a rolling gesture may be applied to objects in a userinterface other than a volume control object. For example, a rollinggesture may be used to adjust a variety of settings in the host system1306 (e.g., channels, brightness, contrast, hue, saturation, etc.). Ifbuttons corresponding to these settings (e.g., a channel button, abrightness button, a contrast button, a hue button, a saturation button,etc.) are present on the multi-dimensional pointing device 1302, thesesettings may be adjusted without having to use a cursor controlled bythe multi-dimensional pointing device 1302 to select correspondingobjects in the user interface 1308 of the host system 1306. Furthermore,a rolling gesture may be used to rotate (or otherwise manipulate oradjust parameters of) objects such as pages of a document, photographs,and the like.

Note that although this specification is described with respect to arolling gesture performed about a longitudinal axis (i.e., a roll axis)of the multi-dimensional pointing device 1302 and a correspondingrolling gesture metric, the embodiments described herein may be appliedto a rotation about a pitch axis or a yaw axis and a corresponding pitchgesture metric or a yaw gesture metric, respectively. These axes areillustrated in FIG. 17, which is a block diagram 1700 illustrating axesof rotation for the multi-dimensional pointing device 1302, according tosome embodiments. As illustrated in FIG. 17, the multi-dimensionalpointing device 1302 has a roll axis 1702, a yaw axis 1704, and a pitchaxis 1706.

FIG. 14 presents a block diagram 1400 illustrating parameters that aretransmitted between the multi-dimensional pointing device 1302 and thehost system 1306, according to some embodiments. As described above, themulti-dimensional pointing device 1302 includes one or moremulti-dimensional accelerometers and one or more multi-dimensionalmagnetometers that are used to determine the attitude of themulti-dimensional pointing device 1302 (e.g., using theattitude-determination techniques discussed above). In some embodiments,firmware 1402 (executed by the one or more processors 1202 of themulti-dimensional pointing device 1200, 1302) processes the attitude andthe button presses to produce corresponding values for dx (i.e., achange in the x position), dy (i.e., a change in they position), ω(i.e., the roll rate or the change in roll angle), and, optionally, α(i.e., the roll acceleration or the time derivative in the roll rate).The dx, dy, ω, and α values represent changes in the pointing device's1302 attitude (also called displacements) between two consecutivesamples in time.

The firmware 1402 then transmits (e.g., using a transmitter circuit inthe multi-dimensional pointing device 1302) values for dx, dy, ω, α, andany button presses to the host system 1306. In some embodiments, a mousedriver 1404 in the host system 1306 receives the values for dx, dy, ω,and the button presses and produces a position, angle or other signal orvalue (e.g., a number of scroll wheel clicks and a scroll wheeldirection (e.g., up or down)) that is used by an application 1406 tomanipulate an object in the user interface. Typically, the firmware 1402is configured to periodically transmit the dx, dy, ω, and button presses(if any), and optionally α, to the host system 1306, at a ratesufficient to provide the illusion of continuous or smooth movement ofthe cursor and/or other object on the user interface. In someembodiments, the firmware 1402 is configured to periodically transmitthe dx, dy, ω, α, and button presses (if any) to the host system 1306 Ntimes per second, where N is a value in the range 2 to 20 (e.g., N=5).

FIG. 15 is a flow diagram of a method 1500 for detecting performance ofa rolling gesture using a multi-dimensional pointing device, accordingto some embodiments. The multi-dimensional pointing device detects(1502) initiation of a gesture by a user of the multi-dimensionalpointing device. In some embodiments, initiation of the gesture by theuser of the multi-dimensional pointing device is detected by detectingthe pressing of a button on the multi-dimensional pointing device. Insome embodiments, the button is selected from the group consisting of: avolume button, a channel button, a video input selection button (e.g., abutton that selects between video sources such as HDMI video sources,component video sources, composite video sources, etc.), an audio inputselection button (e.g., a button that selects between video sources),and a gesture button (e.g., a button that is used when initiating and/orwhile performing a gesture). In some embodiments, detecting initiationof the gesture includes receiving a message from a client computersystem that indicates that the user of the multi-dimensional pointingdevice selected a user interface element (e.g., a menu item, an icon,etc.) in the user interface of the client computer system that initiatesthe detection of gestures. When the detection of gestures is performedon the client computer system, a module of the client computer systemthat determines gestures (e.g., the gesture determination module 1215 inFIG. 12) receives the message.

The multi-dimensional pointing device then detects (1504) performance ofa rolling gesture comprising rotation of the multi-dimensional pointingdevice about a longitudinal axis of the multi-dimensional pointingdevice. In some embodiments, performance of a rolling gesture isdetected when the amount of rotation (e.g., change in rotation angle orrotation rate) of the pointing device exceeds a predefined threshold,while in other embodiments performance of a rolling gesture is detectedwhen the amount of rotation of the pointing device exceeds the thresholdfor at least M consecutive time periods, where M is predefined valueequal to 2 or more (M≧2). Furthermore, in some embodiments detection ofa rolling gesture is independent of the amount of lateral and verticalmovement (dx, dy) of the pointing device, while in other embodiments arolling gesture is detected only when the amount of lateral and verticalmovement (dx, dy) of the pointing device is less than a threshold amount(e.g. only when dx²+dy²≦D, where D corresponds to the threshold).

Next, if performance of a rolling gesture has been detected (1505-Yes),the multi-dimensional pointing device determines (1506) a correspondingrolling gesture metric. The rolling gesture metric corresponds to achange in attitude of the pointing device upon initiation of the rollinggesture. In some embodiments, the rolling gesture metric is selectedfrom the group consisting of a roll angle, a roll rate, and a rollacceleration. In some embodiments, the rolling gesture metric is acombination of two or more of the roll angle, roll rate, and rollacceleration (e.g., a predefined linear combination or predefinednon-linear combination of two or more of the aforementioned values).

If performance of a rolling gesture has not been detected (1505-No),then there is no corresponding rolling gesture metric, and the processwaits until a next sample time (1510) to resume at 1504. As noted above,in some embodiments the multi-dimensional pointing device is configuredto repeat process 1500 two to twenty times per second (corresponding to2 to 20 sample times per second). In some embodiments, if sufficienttime passes (e.g., more than a predetermined period of time, such as 1or 2 seconds, or more than a predetermined number of sample periods)without performance of a rolling gesture being detected, performance ofthe process 1500 is suspended. Process 1500 resumes when themulti-dimensional pointing device detects (1502) initiation of a gestureby a user of the multi-dimensional pointing device.

The multi-dimensional pointing device then conveys (1508) informationcorresponding the rolling gesture metric to a client computer system,where the client computer system is configured to manipulate an objectin a user interface of the client computer system in accordance with therolling gesture metric. In some embodiments, the multi-dimensionalpointing device includes a transmitter circuit configured to transmitdata to the client computer system. After conveying the information tothe client computer system, at least a portion of process 1500 (e.g.,operations 1506 and 1508, or operations 1504-1508) are repeated at anext sampling time.

In some embodiments, operations 1504 and 1505, detecting performance ofa rolling gesture, are not performed. Instead, for every consecutivesampling period during which the point device is active, a rollinggesture metric is determined (1506) and conveyed (1508) by the pointingdevice to the host system. The rolling gesture metric is then evaluatedat the host system to determine if any corresponding action is requiredat the host system. Furthermore, in some embodiments, if the rollinggesture metric indicates either no rotation of the pointing device, oran amount of rotation that is less than a threshold amount, the hostsystem does not rotate or update any user interface objects based on thereceived rolling gesture metric. Thus, in these embodiments, the job ofthe multi-dimensional pointing device is to periodically report changesin its attitude to the host system, regardless of whether those changesin attitude correspond to a rolling gesture or any other gesture.Alternatively, in some other embodiments in which operations 1504 and1505 are not performed, a rolling gesture metric is determined (1506)and conveyed (1508) by the pointing device to the host system only ifinitiation of a gesture by the user of the multi-dimensional pointingdevice has been detected (1502).

Attention is now directed to FIG. 16, which is a flow diagram of amethod for calculating (1506) the corresponding rolling gesture metric,according to some embodiments. The multi-dimensional pointing devicecalculates (1602) a change in attitude of the multi-dimensional pointingdevice since a prior sampling time (i.e., since a prior measurement ofthe pointing device's attitude), based on one or more accelerometermeasurements from one or more multi-dimensional accelerometers of themulti-dimensional pointing device and one or more magnetic fieldmeasurements from one or more multi-dimensional magnetometers of themulti-dimensional pointing device. Next, the multi-dimensional pointingdevice determines (1604-Yes) that the multi-dimensional pointing deviceis undergoing a rotation about a longitudinal axis of themulti-dimensional pointing device based on the change in attitude of themulti-dimensional pointing device. The multi-dimensional pointing devicethen calculates (1606) the rolling gesture metric based on the change inattitude of the multi-dimensional pointing device. If themulti-dimensional pointing device determines (1604-No) that themulti-dimensional pointing device is not undergoing a rotation about alongitudinal axis of the multi-dimensional pointing device, then therolling gesture metric is either not calculated, or is set to apredefined null value (e.g., zero) (1608).

In some embodiments, operations 1604 and 1608, determining whether themulti-dimensional pointing device is undergoing a rotation about alongitudinal axis of the multi-dimensional pointing device, are notperformed. Instead, the rolling gesture metric is always computed (1606)based on the change in attitude (if any) of the multi-dimensionalpointing device. In these embodiments, the rolling gesture metric isthen evaluated at the client computer system (host system) to determineif any corresponding action is required at the host system.

In some embodiments, the rolling gesture is mapped by the clientcomputer system to a rotation operation that is performed on the objectin the user interface of the client computer system. In someembodiments, the object is selected from the group consisting of a dial,a photograph, and a page of a document. Note that the rolling gesturemay be performed on other objects.

In some embodiments, the rolling gesture is mapped to a scrollingoperation that is performed on the object in the user interface of theclient computer system. In some embodiments, the object is selected fromthe group consisting of a web page, a document, and a list. Note thatthe scrolling operation may be performed on other objects.

In some embodiments, the rolling gesture metric is mapped by the clientcomputer system (host system) to a number of clicks of a mouse wheelover a time interval.

Note that the discussion of FIGS. 15 and 16 refers to themulti-dimensional pointing device performing the operations illustratedin FIGS. 15 and 16. However, as discussed above, at least a portion ofthe operations may be performed by the host system. For example, themulti-dimensional pointing device may transmit raw accelerationmeasurements and magnetometer measurements and button presses to thehost system so that the host system can calculate the attitude of themulti-dimensional pointing device and any gestures that are beingperformed by the user of the multi-dimensional pointing device.Similarly, the multi-dimensional pointing device may transmit anattitude of the multi-dimensional pointing device and button presses tothe host system so that the host system can detect any gestures that arebeing performed by the user of the multi-dimensional pointing device.

Note that the methods 800, 1000, 1500, and 1506 may be governed byinstructions that are stored in a computer readable storage medium andthat are executed by one or more processors of a pointing device or ahost system. As noted above, in some embodiments these methods may beperformed in part on a pointing device and in part on a host system.Each of the operations shown in FIGS. 8, 10, 15, and 16 may correspondto instructions stored in a computer memory or computer readable storagemedium. The computer readable storage medium may include a magnetic oroptical disk storage device, solid state storage devices such as Flashmemory, or other non-volatile memory device or devices. The computerreadable instructions stored on the computer readable storage medium arein source code, assembly language code, object code, or otherinstruction format that is interpreted by one or more processors.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method, comprising: at an electronic devicethat includes one or more processors and memory storing one or moreprograms, the one or more processors executing the one or more programsto perform the operations of: detecting a button press of a respectivebutton of a plurality of buttons that are associated with differentoperations, wherein the plurality of buttons include a first button thatcorresponds to a first type of operation and a second button thatcorresponds to a second type of operation; determining, in conjunctionwith detecting the button press of the respective button, a rollinggesture metric corresponding to performance of a rolling gesturecomprising rotation of the electronic device about a longitudinal axisof the electronic device; and after determining the rolling gesturemetric: in accordance with a determination that the respective button isthe first button, initiating performance, in a respective userinterface, of an operation of the first type in accordance with therolling gesture metric; and in accordance with a determination that therespective button is the second button, initiating performance, in therespective user interface, of an operation of the second type inaccordance with the rolling gesture metric.
 2. The method of claim 1,wherein the first button and the second button are physical buttons. 3.The method of claim 1, where initiating performance of an operation of arespective type that is associated with a respective setting includes:determining a direction of rotation of the electronic device about thelongitudinal axis; in accordance with a determination that the rotationis in the clockwise direction, changing the setting in a firstdirection; and in accordance with a determination that the rotation isin the counterclockwise direction, changing the setting in a seconddirection that is different from the first direction.
 4. The method ofclaim 1, wherein the first button is selected from the group consistingof: a volume button corresponding to a volume change operation; achannel button corresponding to a channel change operation; a videoinput button corresponding to a video input change operation; an audioinput button corresponding to an audio input change operation; and agesture button.
 5. The method of claim 1, wherein: the respective userinterface does not include a user interface object associated with thefirst type of operation; and the respective user interface does notinclude a user interface object associated with the second type ofoperation.
 6. The method of claim 1, wherein the rolling gesture metriccorresponds to a change in attitude of the electronic device afterdetecting the button press of the first button.
 7. The method of claim1, wherein the rolling gesture metric is selected from the groupconsisting of: a roll angle; a roll rate; and a roll acceleration. 8.The method of claim 1, wherein determining the corresponding rollinggesture metric comprises: calculating a change in attitude of theelectronic device, corresponding to rotation about the longitudinal axisof the electronic device, based on one or more accelerometermeasurements from one or more multi-dimensional accelerometers of theelectronic device and one or more magnetic field measurements from oneor more multi-dimensional magnetometers of the electronic device; andcalculating the rolling gesture metric based on the change in attitudeof the electronic device.
 9. The method of claim 1, wherein determiningthe corresponding rolling gesture metric comprises: calculating a changein attitude of the electronic device based on one or more accelerometermeasurements from one or more multi-dimensional accelerometers of theelectronic device and one or more magnetic field measurements from oneor more multi-dimensional magnetometers of the electronic device;determining that the electronic device is undergoing a rotation aboutthe longitudinal axis of the electronic device based on the change inattitude of the electronic device; and calculating the rolling gesturemetric based on the change in attitude of the electronic device.
 10. Themethod of claim 1, wherein the first type of operation corresponding tothe first button is a scrolling operation that is performed on an objectin the respective user interface.
 11. The method of claim 10, whereinthe object is selected from the group consisting of: a web page; adocument; and a list.
 12. The method of claim 1, wherein the first typeof operation corresponding to the first button is a rotation operationthat is performed on an object in the respective user interface.
 13. Themethod of claim 12, wherein the object is selected from the groupconsisting of: a dial; a photograph; and a page of a document.
 14. Asystem for detecting performance of a rolling gesture using anelectronic device, comprising: one or more processors; memory; and oneor more programs, wherein the one more programs are stored in the memoryand configured to be executed by the one or more processors, the one ormore programs comprising instructions to: detect a button press of arespective button of a plurality of buttons that are associated withdifferent operations, wherein the plurality of buttons include a firstbutton that corresponds to a first type of operation and a second buttonthat corresponds to a second type of operation; determine, inconjunction with detecting the button press of the respective button, arolling gesture metric corresponding to performance of a rolling gesturecomprising rotation of the electronic device about a longitudinal axisof the electronic device; and after determining the rolling gesturemetric: in accordance with a determination that the respective button isthe first button, initiate performance, in a respective user interface,of an operation of the first type in accordance with the rolling gesturemetric; and in accordance with a determination that the respectivebutton is the second button, initiate performance, in the respectiveuser interface, of an operation of the second type in accordance withthe rolling gesture metric.
 15. The system of claim 14, wherein thefirst button and the second button are physical buttons.
 16. The systemof claim 14, where initiating performance of an operation of arespective type that is associated with a respective setting includes:determining a direction of rotation of the electronic device about thelongitudinal axis; in accordance with a determination that the rotationis in the clockwise direction, changing the setting in a firstdirection; and in accordance with a determination that the rotation isin the counterclockwise direction, changing the setting in a seconddirection that is different from the first direction.
 17. The system ofclaim 14, wherein the first button is selected from the group consistingof: a volume button corresponding to a volume change operation; achannel button corresponding to a channel change operation; a videoinput button corresponding to a video input change operation; an audioinput button corresponding to an audio input change operation; and agesture button.
 18. A non-transitory computer readable storage mediumstoring one or more programs, the one or more programs comprisinginstructions, which when executed by an electronic device, cause thedevice to: detecting a button press of a respective button of aplurality of buttons that are associated with different operations,wherein the plurality of buttons include a first button that correspondsto a first type of operation and a second button that corresponds to asecond type of operation; determining, in conjunction with detecting thebutton press of the respective button, a rolling gesture metriccorresponding to performance of a rolling gesture comprising rotation ofthe electronic device about a longitudinal axis of the electronicdevice; and after determining the rolling gesture metric: in accordancewith a determination that the respective button is the first button,initiating performance, in a respective user interface, of an operationof the first type in accordance with the rolling gesture metric; and inaccordance with a determination that the respective button is the secondbutton, initiating performance, in the respective user interface, of anoperation of the second type in accordance with the rolling gesturemetric.
 19. The non-transitory computer readable storage medium of claim18, wherein the first button and the second button are physical buttons.20. The non-transitory computer readable storage medium of claim 18,where initiating performance of an operation of a respective type thatis associated with a respective setting includes: determining adirection of rotation of the electronic device about the longitudinalaxis; in accordance with a determination that the rotation is in theclockwise direction, changing the setting in a first direction; and inaccordance with a determination that the rotation is in thecounterclockwise direction, changing the setting in a second directionthat is different from the first direction.
 21. The non-transitorycomputer readable storage medium of claim 18, wherein the first buttonis selected from the group consisting of: a volume button correspondingto a volume change operation; a channel button corresponding to achannel change operation; a video input button corresponding to a videoinput change operation; an audio input button corresponding to an audioinput change operation; and a gesture button.